Bump formations in binary attractor neural network 
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This paper investigates the conditions for the formation of local bumps in the activity of binary 
attractor neural networks with spatially dependent connectivity. We show that these formations 
are observed when asymmetry between the activity during the retrieval and learning is imposed. 
Analytical approximation for the order parameters is derived. The corresponding phase diagram 
shows a relatively large and stable region, where this effect is observed, although the critical storage 
and the information capacities drastically decrease inside that region. We demonstrate that the 
stability of the network, when starting from the bump formation, is larger than the stability when 
starting even from the whole pattern. Finally, we show a very good agreement between the analytical 
results and the simulations performed for different topologies of the network. 
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I. INTRODUCTION 

The bump formations in recurrent neural networks 
have been recently reported in several investigations con- 
cerning linear-threshold units 0, 0] , binary units 0, \~ 
Small- World networks of Integrate and Fire neurons 
and in a variety of spatially distributed neural networks 
models with excitatory and inhibitory couplings between 
the cells HQ. 

As has been shown, the localized retrieval is due to 
the short-range connectivity of the networks and could 
explain the behavior in structures of biological relevance 
as the neocortex, where theprobability of connections 
decreases with the distance [8j. 

In the case of linear-threshold neural network model, 
the signal-to-noise analysis has been recently adapted 
0, Q to spatially organized networks and has shown that 
the retrieval states of the connected network have non- 
uniform activity profiles when the connections are short- 
range enough, even without any spatial structure of the 
stored patterns. The increase of the gain or the satura- 
tion level of neurons enhance the level of localization of 
the retrieval states and do not lead to a drastic decrease 
of the storage capacity in these networks even for very 
localized solutions 0- 

An interesting investigation of the spontaneous ac- 
tivity bumps in Small- World networks (SW) [jj, °f 
Integrate-and Fire neurons [5j, has recently shown that 
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the network retrieves when its connectivity is close to the 
random and displays localized bumps of activity, when its 
connectivity is close to the ordered. The two regimes are 
mutually exclusive in the range of the parameter gov- 
erning the proportion of the long-range connections on 
the SW topology of Integrate-and-Fire network, while 
the two behaviors coexist in the case of linear-threshold 
and smoothly saturated units. Moreover, it has been 
stated that the transition between localization and re- 
trieval regimes, in the case of SW networks of Integrate- 
and-Fire neurons, can occur at a degree of randomness 
beyond the SW regime and it is not related, in general, 
to the SW properties of the network. 

The bump formations have been also investigated in 
spatially distributed neural network models with excita- 
tory and inhibitory synaptic couplings. Recently it has 
been shown jg that a neural network only with excitatory 
couplings can exhibit localized activity from an initial 
transient synchrony of a localized group of cells, followed 
by desynchronized activity within the group. This activ- 
ity may grow or may be reduced when depression with 
a given size of the frequency of the inputs is introduced. 
It is also very sensitive to the initial conditions and the 
range of the parameters of the network. 

The result of bump formations have been recently re- 
ported by us, 0, H in the case of binary Hebb model for 
associative network. We stated out that these spatially 
asymmetric retrieval states (SAS) can be observed if an 
asymmetry between the learning and the retrieval states 
is imposed. This means that the network is constrained 
to have a different activity compared to that induced by 
the patterns. 

In the present investigation we regard a symmetric and 
distance dependent connectivity for all neurons within an 
attractor neural network (NN) of Hebbian type formed by 
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N binary neurons {Si}, Si S { — 1, l},i = 1, —,N, storing 
p binary patterns r?f,/x € The connectivity be- 

tween the neurons is symmetric = Cj, € {0, 1}, c« = 0, 
where cy = 1 means that neurons i and j are connected. 

We are interested only on connectivities in which 
the fluctuations between the individual connectivity are 
small, e.g. V, Y] j Cy ~ cN, where c is the mean connec- 
tivity. 

The learned patterns are drawn from the following dis- 
tribution: 

W) = ^« - 1) + + 1): 

where the parameter a is the spa rsity of the code [Tit IT^ | . 
We study the Hopfield model |13j : 

H=jjY^Ji i S i S j , (1) 

ij 

with Hebbian learning rule 
1 P 

Jij =-^2 C V (Vi - a ) (Vj - a)- (2) 
%=l 

Further in this article we will work in terms of variables 

In the case of binary network and symmetrically dis- 
tributed patterns, the only asymmetry between the re- 
trieval and the learning states that can be imposed, in- 
dependent on the position of the neurons, is the total 
number of the neurons in a given state. Having in mind 
that there are only two possible states, this condition 
leads to a condition on the mean activity of the network, 
which is introduced by adding an extra term H a to the 
Hamiltonian 

H a =NR(J2S t /N-a). 

i 

This term favors states with lower total activity J^. Si 
that is equivalent to decrease the number of active neu- 
rons, creating asymmetry between the learning and the 
retrieval states. 

For R = 0, this model has been intensively studied 
[Til IT2I Il5| and shows no evidence of spatial asymmetric 
states. 

For R > 0, the energy of the system increases with 
the number of active neurons (Si = 1) and the term H a 
tends to limit the number of active neurons below aN. 

In the present work we show explicitly that the con- 
straint on the network level of activity is a sufficient con- 
dition for observation of spatially asymmetric states in 
binary Hebb neural network. Similar observations have 
been reported in the case of linear-threshold network , 
where in order to observe bump formations, one has to 
constrain the activity of the network. The same is true 
in the case of smoothly saturating and binary networks 
0, when the highest activity level, they can achieve, is 



above the maximum activity of the units in the stored 
pattern. 

In the present paper we discuss in details the bi- 
nary neural network model, giving a complete analytical 
derivation of its properties. We explain the minimal con- 
ditions for the formations of spatially asymmetric states 
in a recurrent network with metrically organized connec- 
tivity. 

We compare the results for three different topologies of 
the network and show interesting multi-bump formations 
in the case of the SW-like topology. Further extension of 
the analysis to the information, carried by the network, 
reveals that the retrieved states can be well recovered 
by a limited amount of information. We show that the 
stability of the network's retrieval, when starting from 
the bump, is larger than the stability, when starting even 
from the whole pattern. This could be useful in practical 
retrieval tasks with minimal information transmitted. 

The paper is organized as follows: In section Analytical 
analysis and using replica theory, we present our detailed 
mean-field analysis in the case of distance-dependent con- 
nectivities. We derive the equations for the correspond- 
ing order parameters (OP) for finite and zero tempera- 
tures. In section Simulations we present the results of 
the simulations, done for the case of different topologies 
of the network, and we compare them with the analytical 
results. In section Discussion we focus on the stability of 
the phase diagram of the network and especially on the 
dependence of the SAS region on the parameters of the 
model. We discuss the behavior of the critical storage ca- 
pacity when bump formations are present, as well as their 
effect on the information capacity of the network. The 
conclusion, drawn in the last part, shows that only the 
asymmetry between the learning and the retrieval states 
is sufficient to observe spatially asymmetric states. 

II. ANALYTICAL ANALYSIS 

For the analytical analysis of the SAS states, we con- 
sider the decomposition of the connectivity matrix cy by 

(k) 

its eigenvectors a\ ': 

k i 

where A& are the corresponding (positive) eigenvalues. 

k (k) 

For convenience we denote bf = a\ y\k, having 

k 

(k) 

We will assume that a { are ordered by their eigenval- 
ues in decreasing order, e.g. for k > I < A/. To get 

(k) n 

some intuition of what a)j look like, we plot in Fig. [T] 
the first two eigenvectors. 

For a wide variety of connectivities, the first three 
eigenvectors approximate: 

4 0) = Vl/N, (3) 
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FIG. 1: (Color online) The components of the first and the 
second eigenvectors of the connectivity matrix for N = 6400, 
c = 320/iV, Ai = 319.8, A 2 = 285.4. The first eigenvector has 
constant components, the second one - sine-like components. 



have into account a finite numbers of overlaps that con- 
dense macroscopically. To this aim we introduce an ex- 
ternal field, conjugate to a finite number of patterns 
i€i}> v ~ lj 2, s, adding to the Hamiltonian the term 

In order to impose some asymmetry in the neu- 
ral network's states, we also add the term H a = 
NR(£iSi/N-a). 

The whole Hamiltonian, we are studying, is: 



» ^E Si^sj + 5> y E era 

ijfi i/— 1 i 

+NR{J2Si/N-a). (6) 



and 



4 1} = y/2/N cos{2Ti\k - k \/N) (4) 



4 2) = \/2/N sin{2n\k - fc |/iV). (5) 



By using the "replica method" [T^, for the averaged 
free energy per spin we get: 

/=-lim lim J-(((Z n ))-l), (7) 

n— >0 N^oo pnJV 



where ((...)) stands for the average over the pattern dis- 
Following the classical analysis of Amit et al. fl5l | . tribution P(£f ), n is the number of the replicas, which 
we use Bogolyubov's method of quasi averages to are later taken to zero and (3 is the inverse temperature. 

I 



The replicated partition function is 



«Z")) = ((Tr S pexp 



JL 

2cN ~*'~' J ^3 ~3> 2c' 



(8) 



v E h " E - E WE S "/ N - «) 

i> i : p p i 

The decomposition of the sites, by using an expansion of the connectivity matrix Cij over its eigenvalues A , I = 1, M 
and its eigenvectors a\, gives: 



((r))=e-^// Trspexp fl^^ Kf ^ )( ^; i 5 ) 

fj.pl ij 

E h " E - E p rn C£ s ii N 1 



(9) 



Introducing variables at each replica p, each configuration and each eigenvalue, we get: 



((Z n )) = e ~Pv n / 2c +P RaN 



rsp /nS exp/3c7V f-sEK) 2 +E<^E^ 

flip \ /ip/ /ip/ i J 

ex P/ 3civ l-^EK/) 2 + E E er^i + ^ E^" 5 ' + ^f) J )), 



(10) 



where we have split the sums over the first s-patterns and the remaining (infinite) p — s ones. 
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The "condensed" order parameters m pl have a clear 
physical meaning as an overlap between the pattern v 
and the state of the neuron in the replica p of the sys- 
tem, modulated by the Z-th eigenvalue of the connectivity 
matrix: 

In the present investigation, the main physical hypothesis 



is that only a finite number, k, of the order parameters 
m u p j, I = 1, k are macroscopically different from zero. 
As we can see later in this paper the simulations also 
confirm this hypothesis. This avoids the introduction of 
infinite number of parameters, dependent on the state of 
the system, that would make impossible the analytical 
solution of the problem. 

After taking the averages over the pattern, for the first 
term we get: 



exp/3 ( -\ ]T(m£) 2 + £ mWtfSfffl ] . (12) 

jxpl perlkiy, 



The integration of the last expression over the OP gives: 



Upl 



2tt 



J J] dq% exp (- P -Tr \n[A%]) ]J 5(q% - -L £ SfS^). 

palk pcrlk i 



(13) 



The expression within the (5-function can be regarded as 
a definition of the OP q l pa . 

Now let us suppose that the order parameter q l pa 
can split as a product of two terms: one, which only 
depends on the replica's and the eigenvalue's indexes 
and another one that introduces the spatial dependence 
of the distribution of the eigenvalues and eigenvectors: 



J 



In this way we 



C = Ipa^bW/cN = q^Mh/cN). 
can keep only one index for the OP q l pa . when taking into 
account the spatial distribution of the matrix of interac- 
tions. 



Introducing the parameter r l pal 
the last expression we obtain: 



conjugate to q l , for 



/n 

fipl 



2ir 



pal pal 



expcTV [-\af(l a 2 )J2r l p J P , + ^ a ^^ 1 « 2 ) E^SfSf 6{6{) 

V pal ipal J 



(14) 



where the parameter a = p/N is the storage capacity of the network. 
The matrix A l pa is 



4. = S P a (1 - 0(1 - a 2 )m(l - qi )) - /3(1 - a 2 )q m 



(15) 



and hi = Xi/cN. 
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For the replicated partition function ((Z n )), after taking the limit h? — > 0, we obtain: 

((z n )) - e-^a-^z-^/n^/n^ 



pal 



expcTV ( -| ^(m^) 2 - \TTHA l pa ] ~ - a 2 ) ]T 



Ttsp exp cA~ 



2c7V 



ipalk 



^E^^E^+^E^ 



i /pi 



(16) 



Supposing Replica Symmetry (RS) ansatz, i.e., = m" for any replica index p and = q l ,r l pa . = r; for p ^ a, 



for the free energy, Eq.Q we obtain 
a(l-a 2 ) 



/ = 



2c 



fia+^Trln[^] + i^(mn 2 



a(3(l - a 2 ) ^ , 



E 

Zfc 



n/3 



In Trsp exp 



?-a^(l-a 2 )^r^f 1 9f^ + 



2ciV 



ipal 



^E m ^ E # S ? 6 S + ^E s ? 



V pi 



(17) 



The over-line in the last expression has its clear p hysic al 
meaning as an average over the spatial distribution (.)^ = 

As a next step, let us suppose that the average over a 
finite number of patters can be self-averaged 0|- I n 
our case this is expressed by the following decomposition 
S?S?b\b\ « S p S <7 (b l i ) 2 , which is reasonable to assume as 



a first approximation by taking into account the effect of 
the spatial distribution. This approximation permits to 
complete the analytical analysis. 

After taking the trace Trsp and the limit n — > 0, in 
Eq. HI 7(1 and using the saddle point method, we end up 
with the following expression for the free energy: 



f 1 fi 2^,!^/ l2 a(3(l-a 2 ) \ - a/?(l-a 2 ) 

k k 

+ ^EHl-/?(l-« 2 K+/3(l-a 2 )%)- 
Ap k 

- 0(1 - a 2 )q k {l - 0(1 - a 2 H + /3(1 - a 2 )*)" 1 ] " 



^ Hkr k 



(18) 



1 r dze-* 2 / 2 



0J l2:r 

The equations for the OP r^, and qfc are respectively: 

q k (l-a 2 ) 



In 2 cosh/3 z Ml-a 2 )^n6^ + ^m^+i?&° 



'A' 



(1-/J(l- (ft))' 



(19) 



mi, = I y^^t^nhp \ zja(l - a 2 )J2nb[b\ + Y,m^ t b[+ Rb° 



'2tt 



(20) 



6 



and 



Ik 



dze~ z2 ' 2 



2n 



(b k ) 2 tanh 2 P z a(l - a 2 ) E nb\b\ + E m,&&{ + Rb<] 



(21) 



The following analysis refers to the case T = 0. Keeping Ck = 0(fJ,k ~ Qk) finite and limiting the above system only 
to the first two m^-s, the Eqs. (|19I21|) read: 



Here 



m 



nil 



1 - a 

47T 



2 



g{(j>)d(j) 



'2m 



-1-a 

47T 



2 



g(4>) sin <fi dip 



i r 

Co = — / g c {4>)d(j) 

27r J-7T 

5c (^)sin 2 0d0 

-7T 



Mfc(l - a 2 ) 

[1 - (1 - a 2 )C k ] 



g(4>) = erf(xi) + erf(x 2 ) 

5c (0) = [(i + a )e-^) 2 + (i- a ) e -^) 2 ]/[^y] 

xi = [(1 — a)(m + 7711-^/2^1 sin0) + R]/y 

%2 = [(1 + a)(m + 7771-^/2/^1 sin0) - R]/y 

y = \/2a(l - a 2 )(r Q + 2fi 1 (r 1 - 1 + a 2 ) sin 2 , 



(22) 
(23) 
(24) 
(25) 
(26) 



(27) 
(28) 
(29) 
(30) 

(31) 



For values of the order parameters mi = 7712 = ... = 
and R = 0, we obtain a result similar to that of Canning 
and Gardner . The classical result of Amit et al 
is obtained when m = 0. 



The result of the numerical solution of Eas. l|22l26|l is 
show in Fig|2{c,d), where we have rescaled the order pa- 
rameters mo and 777i to belong to the interval [0,1], in- 
stead of the interval [0, 1 — a], for any sparsity a. The 
sharp bound of the phase transition is a result of tak- 
ing into account just two terms of m^, k — 0, 1 and the 
lack of finite-size effects in the thermodynamic limit. The 
corresponding behavior of mo and mi , obtained by simu- 
lation, is presented in the top panels of Fig. |2fa,b). The 
good correspondence between the numerical result and 
the simulations confirm our hypothesis of the finite num- 
ber of macroscopic OP m/ 7^ 0. The physical reason for 
this suggestion lies on the fact that the bump formations 
are compact structures with a large size and therefore 
the low-frequency components, e.g those corresponding 
to large eigenvalues of the matrix of connectivities, are 
responsible for the observed behavior. 



III. SIMULATIONS 

We performed simulations for a network in form of a 
circular ring, with a distance between the neurons i and 

3- 

\i — j\ = mm(i — j + N mod N, j — i + N mod N), 

using three different topologies, explained in this section. 

To compare with the results of Ref.[l|, we work with 
a typical connectivity distance a x N . This Gaussian- like 
distribution defines the first topology we are studying. 



P(cy = 1) = C 



1 



2TT(T r N 



Po 



(32) 



where po is chosen to normalize the expression in the 
brackets. When a x is small enough, then spatial asym- 
metry is expected. 

The dynamics of the network at time t + 1 and tem- 
perature T = for our case is 

S t (t + 1) = sign ^ E E ggcisMt) T,)j , 
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FIG. 2: (Color online) Simulation (a) and computation of mo, mi according to Eqs, 1221261 (c). The sparsity of the code is 
a = 0.8 and R — 0.57. The form of the bump, represented by its local overlap, Ea, l33H . according to the simulation (b) and 
the theory - (d) bold line. Parameters of the simulation: p/(cN) = 0.1, N = 6400, c = 0.05, a x = 500, m(i) smoothed with 
lengths 100 and 500; fii = 0.95. 



where T), is the threshold of the system, which is in gen- 
eral nonzero, due to the extra energy term H a . 

If the retrieval state, corresponding to the pattern 
is Si, the mean overlap is mo = J2i an d the local 

overlap at site i is 



m{i) = &S t 



(33) 



In the case when Si follow a single sine wave, the ideal 
measure of spatial asymmetry would be 



1 

TOl = — 

N 



E 



ikSke 



which corresponds to the theoretically derived OP mi 
from Eqs.|Q3EDj!. 

Simulations with more sharp localized connectivity 



P(Cij = 1) oc 



1 — 6 cos ip 
l-2bcos(p + b 2 ' 



(34) 



with if = 2ir\i — j\/N and b being some parameter, show 
similar results. This connectivity defines the second type 
of topology we are studying. It has the advantage that 
the eigenvectors of the connectivity matrix are cosine 
waves and the eigenvalues are known. The results of 
the simulations for different a and a are shown in Figs. 
EEI These two topologies, Eas. (|32l34|) . give very similar 
results. 



If R = 0, no asymmetry can be observed for any a xi 
up to the level of the network fragmentation (Fig. |3J. 
No difference between asymmetric and symmetric con- 
nectivity is observable for any connectivity c < 0.05 and 
any of the topologies tested. 

The sparse code increases the SAS effects: Fig0Jb), vs. 
Fig|2fa), but SAS can not be observed for any sparsity 
a if the proportion of the firing neurons is kept to be 
equal to a , i.e. when the retrieval and the memorized 
states have the same level of activity. However, even 
without sparsity (a = 0) the bump states can be observed 
FigHb). 

One could expect that the bump formations actually 
will deteriorate the performance of the neural networks, 
because the bump is localized in a limited part of the 
network and this effectively shrinks the active part of 
the network. However, it results not to be so. Namely, 
if we start the retrieval process from the bump tuq < 1, 
fixing R in such a way that a bump will be produced, this 
retrieval is better than the retrieval achieved, starting 
from totally overlapping pattern tuq = 1. 

In the case of one-dimensional network given with 
Gaussian-like distribution of the connectivity on a cir- 
cular ring, we observe the behavior, given in Fig[3J 

In Fig|S]we observe a pronounced stability of the posi- 
tion of the center and the boundaries of the bump, when 
using the bump initial condition, while in the case of us- 
ing the whole pattern as an initial condition, this is no 
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FIG. 3: (Color online) Binary attractor networks with the 
same distribution of the pattern and the retrieval activity 
(R=0). The overlap m and the power of its first Fourier 
transform are chosen as a measure of the existence of SAS. 
The panel (a) is for TV = 6400, c = 80/7V, a x = 100, a = 0.8, 
the panel (b) is for N = 6400, c = 160/A, a x = 200, a = 0.8. 
None of the networks present SAS. The same is true for sparse 
code, different dilution and other topologies as for example 
the defined by Eqs. • 



longer true. In the last case, one observes a shrinking of 
the region where the SAS state is observed, given by the 
rapid decrease of the order parameter (red dashed line), 
which occurs for smaller values of the load compared to 
the case of initial bump attractor. 

This behavior is repeated for three different values of 
the bump initial conditions too = 0.15, too = 0.2 and 
too = 0.3, given in Figs[5ja,b,c). The corresponding be- 
haviors when starting from the whole pattern are given 
in FigsGJd,e,f). 

Figs. El also show that the difference between the two 
behaviors is smaller when the size of the initial bump is 
smaller. For example, when an initial bump condition 
is mo = 0.3, then the region of stability of the solutions 
extends up to value of the load a = 0.35, while it de- 
creases up to a = 0.29 for uniform initial condition. For 
mo = 0.15, these values are a = 0.25 and a = 0.2 respec- 
tively. 

The last result clearly points out the role of the bump 
formations on the stability during the retrieval. 

The comparison of the overlap in the case of initial con- 
ditions corresponding to bump of size too = 0.15, uniform 
distribution too = 1 and uniform initial condition with 
to = 0.15 is given in Fig El 



FIG. 4: (Color online) Spatial asymmetry observed by its first 
Fourier component power mi with symmetric code (a = 0). 
The panel (a) shows the network with N = 6400, c = 0.05, 
a x = 500 and R = 0. The panel (b) shows the network with 
N = 6400, c = 0.05, a x = 500 and R = 0.5. SAS is observed 
only when R ^ 0. 



From FigEJit can be observed that the uniform initial 
condition with to = 0.15 gives the worst result for the 
size of the retrieval solution. The overlap shrinks to zero 
for load less than a = 0.2, while this value is a — 0.29 
when starting form the bump. We face again with the 
conclusion that the bumps are stable formations and en- 
hance the performance of the network. 

This result could serve as a basis of the biological rel- 
evance of the bump formations for the whole stability 
of the network to learn and to retrieve information effi- 
ciently. 

Finally, we also performed simulations in the case of 
the small- world (SW) topology |gL fTo| with re- wiring rate 
we [0,1]: 

P(cy = 1) = (1 - w)0(c -\i- j\/N) + LJC, (35) 

where 8 is the theta function. For R = 0, no SAS behav- 
ior is observed. 

However, when R ^ 0, the simulations show more com- 
plex behavior with the appearance of several bump for- 
mations, as well as phases with no bumps but significant 
m ii FigEl It seems probable that the multiple bumps are 
due to the roughness of the SW eigenvalues distribution 
. This different behavior of the S W topology deserves 
future attention. 
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FIG. 5: (Color online) The dynamics of the bump formations. On the vertical axis the center and the boundaries of of the bump, 
as well as the asymptotic value of m are given as a function of the load a for three different initial conditions mo = 0.15, mo = 1 
(Fig. a and d), mo = 0.2, mo = 1 (Fig.b and e) and finally mo = 0.3, mo = 1 (Fig.c and f). More stable behavior is observed 
when starting from the bump (Fig.a,b,c), than starting from uniform initial condition (the pattern itself) (Fig.d,e,f). The 
parameters of the simulations are N = 6400, c = 0.05 and a x = 500. 
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FIG. 6: (Color online) The behavior of the overlap for three 
values of the initial conditions: starting form the bump forma- 
tion with mo = 0.15, (Fig. a), from the pattern itself (uniform 
distribution mo = 1), (Fig.b) and from uniform distribution 
with mo = 0.15, (Fig.c). A clear advantage of the first ini- 
tial condition is observed with respect to the the size of the 
region of retrieval. The parameters of the simulations are 
N = 6400, c = 0.05 and a x = 500. 
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IV. DISCUSSION 

The numerical analysis of Ecis. H22I26|) gives a stable re- 
gion for the solutions corresponding to bump formations 
for different values of the load a, the sparsity a and the 
retrieval asymmetry parameter i?, shown in Fig[S] As 
can be seen, the sparsity of the code a enhances the SAS 
effect, although it is also observed for a = 0. 



FIG. 7: (Color online) Smoothed local activity for small-world 
connectivity. In the panel (a) no bumpiness, but significant 
fluctuations are present (mi = O.lmo). In the panel (b) single 
bump is presented. And the last panel (c) shows multiple 
bump activity. N = 6400, c = 0.05 and a = 0.6(a), a = 0.7(b) 
and a = 0.8 (c), re- wire rate u — 0.06, R = 0.5. The same 
effect can be achieved by changing (c,lo). 
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FIG. 8: (Color online) Phase diagram R versus a for a = 0.001 
(a/c = 0.02). The SAS region, with m / 0, mi ^ is 
relatively large and decreases by increasing the load (bottom) . 
Note that not all of the values of a are feasible with arbitrary 
mean connectivity of the network c. 
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FIG. 9: (Color online) The critical storage capacity a c for 
a = 0.4. A drop of the capacity is observed on the transi- 
tion between normal retrieval state and state with localized 
activity. 



As we expected, the asymmetry factor R between the 
stored and retrieved patters is very important in order to 
have spatial asymmetry. The diagram in FiglHl shows a 
clear phase transition with R. For small values of R, the 
effect (SAS) is not present. If R > 1, then the only stable 
state is the trivial Z state, as all the nonzero solutions 
are suppressed. Only for intermediate values of i?, the 
bump solution exists. 




FIG. 10: (Color online) The bumpiness B of the network for 
different values of R and a. 



When increasing the load, the area corresponding to 
SAS states shrinks as is shown in Fig(H[b). This is due 
primary to the fact that the critical capacity a c of the 
network drops when the system enters into the SAS state. 
More precisely, the behavior of the critical storage capac- 
ity as a function of the asymmetry parameter R, shown 
in FigE| presents a drastic drop of its value at the tran- 
sition from homogeneous retrieval (symmetric) state to 
spatially localized (asymmetric state). Effectively, only 
the fraction of the network in the bump can be excited 
and the storage capacity drops proportionally to the size 
of the bump. For the values of the sparsity of the code 
a = 0.4, where the effect of the asymmetry parameter is 
maximal, the decrease of the critical capacity is approx- 
imately twofold. 

We have also investigated the effect of the parameters 
R and a on the bump formation. For this, we introduce 
a parameter that can give a criterion for the bumpiness 
as the relative ratio between the two order parameters 
mo and mi, given by Ecis. l22l2tj|) : 



B 



mi' 



mo/ 



m,\ i 



The behavior of B is presented in Fig^B It is seen that 
the simultaneous increase of both R and a inside the area 
where the SAS behavior is observed, favors the bumpi- 
ness in the binary network. This behavior is due to the 
stronger localization of the activity as the extra term H a 
favors states with lower total activity. From the other 
side, increasing a makes the code sparser, which favors 
the metrical organization of both patterns and neurons. 

Finally we have investigated the information capacity 
of the network. For this aim we calculated the corre- 
sponding expression for the mutual information |20| in 
the case of order parameters mo and mi : 
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FIG. 11: (Color online) (a) The information capacity versus 
the asymmetry retrieval factor R for different values of the 
sparsity a. (b) mo, mi for a = 0.2. The inside figure in (a) 
shows the behavior of the information capacity near the point 
where it sharply decreases to zero. 



tion capacity is very small although non zero. If from 
one side, the information capacity Im /cN, measures the 
number of bits, needed to represent the state and if from 
the other side, the overlap parameters (mo, mi) measure 
the degree to which one learned state can be recovered, 
then having small information capacity and large OP 
mo, mi means that the original state can be recovered 
well with very small amount of information. This result 
could be useful for effective retrieval of patterns by using 
very small amount of information, especially in combi- 
nation with the observed improvement of the stability of 
the network, provided that the retrieval starts from the 
bump. 

Similar results have been reported in with respect 
to the increase of the computational power of the struc- 
tured network. The last is expressed by the ability of 
the network to retrieve several patterns, each in a dif- 
ferent location, whose combination could form the global 
combinatorially large pattern. 

Apart of the advantage of the localized states in terms 
of the minimal cost in transferring information, such 
structures could have important biological relevance for 
the cortical modules of the mammalian brain, which are 
characterized by geometrical distribution of the synap- 
tic connections. Recent ex peri ments of the activity of 
a macaque temporal cortex [22] have demonstrated that 
the receptive fields of the visual evoked activity patterns 
are restricted up to three fold, when several patterns are 
present together. The last result seems to be coherent 
with the picture of localized activity in a neural network 
with metric organization of the synaptic activities. 



CONCLUSION 
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Here F[] is the hyper geometric function 21] . The first 
term is the normal expression for the mutual informa- 
tion (the sparsity of the code eliminates the symmetry 
between to and —to). The next two lines of the equation 
can be regarded as a correction to the usual expression 
due to OP toi ^ 0. 

With the aim of Eas. l|22l2(j[) and the above expres- 
sion, for the information capacity Im/cN we obtained 
the behavior, presented in Fig llll which has been also 
reproduced by simulations. This behavior is non trivial 
and shows a well pronounced maximum for intermediate 
values of the sparsity of the code a. The corresponding 
behavior of the order parameters too and mi, presented 
in Fig lllf b). shows that inside the SAS region, their val- 
ues are significantly different form zero, but the informa- 



In this paper we have studied the spatially dependent 
activity in a binary neural network model when retrieval 
and learning states have certain degree of asymmetry. We 
have shown that nor asymmetry of the connections, nei- 
ther sparsity of the code are necessary to achieve bump 
solutions for the activity of the network, but rather dif- 
ferent symmetry between the retrieval and the learning 
states. 

The extension of the classical method of Amit et al. 
[l5| in the case of spatially dependent connectivity has 
permitted, within some approximations, the derivation of 
the above results analytically. The good correspondence 
with simulations, tested for different topologies of the 
network, is also discussed through the paper. 

We have shown that the region of the spatially asym- 
metric states is relatively large in the parameter space 
(R, a) and we have measured quantitatively how the 
bumpincss of the network depends on these two param- 
eters. 

A drop of the critical storage capacity is observed in the 
transition form symmetric to SAS states, which is due to 
the fact that effectively only the fraction of the network in 
the bump can be excited and the storage capacity drops 
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proportionally to the size of the bump. 

When the recovering of the pattern starts from a state 
corresponding to the bump formation, the network be- 
haves in a better way in terms of stability and capacity 
compared to the case of uniform initial conditions. This 
happens even if the overlap with the uniform initial con- 
ditions is significantly larger. The latter result could ar- 
gue the biological relevance of the bump formations for 
effective retrieval of information. 

Finally we have discussed the behavior of the infor- 
mation capacity versus the asymmetry retrieval factor R 
and we have discussed its sharp decrease in the paramet- 
ric region, where bump formations are present. 
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